-
Notifications
You must be signed in to change notification settings - Fork 302
mintmaker: set GOMEMLIMIT to avoid controller OOMKilled crashes #7576
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mintmaker: set GOMEMLIMIT to avoid controller OOMKilled crashes #7576
Conversation
Set GOMEMLIMIT=7GiB so the Go runtime respects Kubernetes memory limits. Although the controller is allocated 8GiB memory, in large clusters (e.g., p02 with 5000+ components) it can still be OOMKilled, possibly due to the Go runtime ignoring container limits.
Code Review by Gemini## Code Review
### `components/mintmaker/production/base/manager_patch.yaml`
**Improvement:**
The addition of `GOMEMLIMIT` is a good practice for Go applications running in containers to help the Go runtime better respect the allocated memory limits and prevent OOMKilled issues. The value of `7GiB` for an `8GiB` limit provides a reasonable buffer for non-Go memory usage.
```diff
--- a/components/mintmaker/production/base/manager_patch.yaml
+++ b/components/mintmaker/production/base/manager_patch.yaml
@@ -8,10 +8,13 @@ spec:
spec:
containers:
- name: manager
resources:
limits:
cpu: 500m
memory: 8Gi
requests:
cpu: 100m
memory: 8Gi
+ env:
+ - name: GOMEMLIMIT
+ value: "7GiB"
|
Since earlier today, the mintmaker controller in the p02 cluster has been repeatedly crashing due to OOMKilled errors. I’m not certain of the exact root cause, but it appears to be related to the Go runtime not respecting container memory limits. I’m setting |
do you think we could log the memory in controller via https://pkg.go.dev/runtime/metrics ? |
I’m currently investigating and analyzing the controller’s memory usage, and I will include this in the research plan |
Awesome, thank you, let's merge this in the meantime |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: qixiang, staticf0x The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
8a0e051
into
redhat-appstudio:main
Set GOMEMLIMIT=7GiB so the Go runtime respects Kubernetes memory limits. Although the controller is allocated 8GiB memory, in large clusters (e.g., p02 with 5000+ components) it can still be OOMKilled, possibly due to the Go runtime ignoring container limits.